The present invention relates to the image display arts. It finds particular application in conjunction with providing three-dimensional presentations of diagnostic medical images on remote video monitors and will be described with particular reference thereto. In particular, the invention relates to interactive visualization of internal cavities in an organism, such as the intestines, bronchi, arteries and the like. However, it is to be appreciated that the invention finds application to other areas, such as virtual viewing of internal cavities of non-organic subjects such as pipes and sealed vessels. In addition, it will be appreciated that the invention has broader application in conjunction with generating three-dimensional diagnostic images from data acquired from other imaging modalities, e.g., by magnetic resonance imaging and ultrasound.
Heretofore, an endoscope is used to view passages through or the interior of organs such as the bronchi, esophagus, stomach, etc. The endoscope is threaded into internal cavities within the human body to provide real-time, high resolution views of the interior. The views may be recorded on video tape for later viewing. Further, the video images may be electronically transmitted to remote locations. However, there are a number of disadvantages with endoscopic examinations and video recordings thereof. The endoscope provides the operator and remote viewers with only a limited field of view without the ability to review in a reverse direction. Another problem with endoscopic examination is that it is not capable of being used with cavities that do not have an opening to the outside. In this regard, where possible, certain cavities must be surgically perforated to allow access to the endoscope. Further, because endoscopic examination can be uncomfortable or even painful, the procedure requires some sedation or anesthesia to reduce patient discomfort.
The use of computed tomography (CT) scanners using X-rays also provides limited inspection of internal cavities. CT scanners irradiate the planar region of a subject from various angles and detect the intensity of radiation passing there through. From the angle and radiation intensity information, two dimensional image representations of the plane are reconstructed. A typical image representation includes a 512 by 512 pixel array, although smaller and larger arrays are known. In a black and white image, each pixel has a corresponding value or number which is indicative of the gray scale to be displayed at that pixel. For three-dimensional imaging, a plurality of slices are generated, e.g., 60 closely adjacent parallel slices, each of which is represented by a 512 by 512 array of pixel values. The pixel values of the multiple slices are treated as a 512 by 512 by 60 pixel array or three dimensions of image data. Various planes or other surfaces can be defined through the three dimensional data and displayed. However, visualizing the interior of a three dimensional cavity from a series of slices is difficult.
In effort to improve visualization, techniques have been developed for generating a three dimensional representation allowing the inspection of an object along any cutting plane. With these techniques, appropriate planes and surfaces may be selected which permit viewing of the internal surfaces of cavities in the human body. That is, a slice image may be generated through a length of the esophagus. Such a slice can be processed to reveal the internal surface of one-half of the internal cylindrical length of the organ. Generally, such three dimensional presentations include a display of only the extended surfaces which a viewer would see and an internal part of the object through the cut of the object by an appropriate plane or surface.
To generate the pixel values for display, every pixel value of the three dimensional data is examined. Each data value is examined to determine whether or not it shows in the resultant image. Each data value which does show is assessed relative to the other data values to determine what contribution, if any, it makes to the image. None can be readily dismissed as not showing. Specifically, air produces a pixel value characteristic of black. Because air is transparent to the viewer, values from pixels hidden behind pixels whose values are indicative of air show through, hence must be displayed. Analogously, other types of tissue that have characteristic pixel values or CT numbers are also defined as transparent and removed from the view. Hence, the location of the pixel within the data alone is not determinative of whether or not the pixel value would show in the image. Rather, each pixel value has to be considered in the context of its surrounding pixels. This is computationally very time-consuming. Note that a 512 by 512 by 60 pixel data set contains almost 16 million pixels. Various techniques have been developed, many of which are application-specific, i.e., for reducing or identifying a subset of all available pixels to project up to the cutting surface or viewing screen to determine their contributions.
To visualize depth, the angle of the tangent to the surface at each point is estimated and shading is added in accordance with the angle of the surface tangent relative to a preselected illumination point. In a black and white CT image, the shading is added by increasing the brightness to whiteness of each pixel value in proportion to how nearly perpendicular it is to the light source, and by increasing the black scale in proportion to the degree that the tangential surface faces away from the light source. For example, a gray scale value that is proportional to the sine/cosine of the angle between the tangent surface and the light source may be combined with each pixel value.
Once the three dimensional presentation is displayed on the screen, it is often advantageous to view it from a different orientation. For example, a critical surface portion may be partially obscured or it may be necessary to see the back side before starting surgery. For the new viewpoint, the entire process is repeated anew. Effectively, all of the data within the three dimensional volume is rotated to the appropriate orientation relative to the viewing screen, and the contribution of each pixel is projected up the plane of the screen for reassessment. All of the data is rotated or shifted to achieve the proper location of the screen relative to the data before the data was projected up to the screen. The shifting of almost 16 million pixels of data and the interpolation of data, where necessary, further adds to processing time.
Viewing the CT data with key-frame techniques provides yet another improvement in the visualization of internal cavities. In one key-frame technique, an operator uses software to move through the data and render images from a certain viewpoint and in a certain direction. The operator generates a series of such images which may then be viewed sequentially as an animation. Some of the problems with key-frame animation are the same as with the video recording of an endoscopy study. The secondary viewer has a limited field of view and is restricted to those key-frames selected by the initial operator. Further, such studies are overly large and cumbersome to use on networks.
Another technique to visualize internal cavities is forward-looking virtual endoscopy which is a type of key-frame animation. Using a software package, an operator selects a path within a cavity or passage. Sequential forward-looking views or key-frames are then generated along the path and displayed. Forward-looking virtual endoscopy also suffers from some of the same problems as actual endoscopy. That is, a secondary viewer is constrained to a limited field of view and only to forward-looking views. In addition, virtual endoscopy studies generate large amounts of data which tax network resources in their transmission, require significant time to download and require significant resources for their storage. In order to view a study prepared from a data set, an entire video sequence must first be downloaded before any image may be viewed. Data compression of a virtual endoscopy video can speed transmission and smooth playback over a network but it also degrades noticeably the quality of the images.
Forward-looking virtual endoscopy may be enhanced by capturing images outside of key-frame animation. In this technique, the radiologist leaves the key-frame animation paradigm of image capture to manually reorient for capture of structures which are, for example, perpendicular to a viewpath through an organ. This also has its problems. First, it is difficult and time-consuming to capture views while constructing a sequence of virtual endoscopy images. Second, some visio-spatial context of the organ is lost in the frequent reorientations necessary for adequate coverage of the organ structure and subsequent reorientations of the camera to recover the original path. Thus, there is a possibility that the radiologist charged with capturing sequences will not capture some views crucial to diagnostic confidence due to the difficulties inherent in manual navigation and capture of images in virtual endoscopy datasets and subsequent reorientations of the rendering camera. In particular, the back surfaces of structures blocking passages require special attention. Failure to capture fully and accurately the context of the structure is possible if capture is left entirely to manual camera orientation.
The present invention contemplates a new and improved method and apparatus for inter-active virtual endoscopy over networks which overcomes the above-mentioned problems and others.
In accordance with the present invention, a medical diagnostic apparatus presents a three dimensional image presentation on a two dimensional display. An image data memory for stores image data indicative of a three dimensional volume. An image data memory accessor selectively accesses the stored image data. A view renderer renders a plurality of views which in total cover the entire visual space about a viewpoint within the three dimensional volume. A view compositor combines the plurality of views into a full image covering the entire visual space about the viewpoint. A mapper maps the full image into a spherical, panoramic image. A video processor displays a portion of the spherical, panoramic image on the two dimensional display.
In accordance with a more limited aspect of the present invention, a pitch control controls the pitch of the displayed spherical panoramic image. A yaw control controls the yaw of the displayed spherical panoramic image. An environmental mapping processor controls the video processor to display other portions of the spherical, panoramic image in accordance with the pitch and yaw controls.
In accordance with another more limited aspect of the present invention, a server between the image data memory accessor and the view renderer permits remote display of the panoramic image on the two-dimensional display.
In accordance with another aspect of the present invention, a method of generating a three dimensional image presentation using a computer is provided. Image data is stored which is indicative of a three dimensional array of pixels. A viewpoint is selected within the three-dimensional array. A plurality of two-dimensional arrays is generated which in total cover the entire spherical space about the viewpoint At least one of the plurality of two-dimensional arrays is divided into a plurality of first polygon arrays. The plurality of first polygon arrays is scaled into a plurality of second polygon arrays. The plurality of second polygon arrays and a portion of the plurality of two-dimensional arrays are combined to form a full two-dimensional array covering the entire spherical space about the viewpoint. The full two-dimensional array is mapped into a spherical view. At least a portion of the mapped, full, two-dimensional array is displayed as image pixels in a human-readable display.
In accordance with a more limited aspect of the invention, the three dimensional array of pixels is generated through a medical diagnostic exam of a patient.
In accordance with a still more limited aspect of the invention, the plurality of two-dimensional arrays which in total cover the entire spherical space about the viewpoint is six. The first polygon arrays are triangular arrays. The second polygon arrays are rectangular arrays.
One advantage of the present invention is that it provides full, unobstructed spherical panoramic views from within a three-dimensional array of image data.
Another advantage of the present invention is that it facilitates remote viewing and diagnoses of internal cavities as represented in the three-dimensional array of image data,
A further advantage of the present invention is that it permits greater diagnostic confidence for secondary viewers of the three-dimensional array of image data.
Still another advantage of the present invention is that it permits rapid transmission of the images to remote locations over networks and more efficient use of network resources.
Further, although an endoscope examination is mostly non-invasive, the procedure still requires some sedation or anesthesia to reduce patient discomfort.
Still further advantages of the present invention will become apparent to those of ordinary skill in the art upon reading and understanding the following detailed description of the preferred embodiments.